- Hello, hi. So I want to get started. Welcome to CS 231N Lecture 11.
We're going to talk about today detection segmentation and a whole bunch of other really exciting topics around core computer vision tasks.
But as usual, a couple administrative notes.
So last time you obviously took the midterm, we didn't have lecture, hopefully that went okay
for all of you but so we're going to work on grading the midterm this week, but as a reminder please don't make any public discussions about the midterm questions or answers or whatever
until at least tomorrow because there are still some people taking makeup midterms today and throughout the rest of the week
so we just ask you that you refrain from talking publicly about midterm questions.
Why don't you wait until Monday? [laughing] Okay, great.
So we're also starting to work on midterm grading. We'll get those back to you as soon as you can, as soon as we can.
We're also starting to work on grading assignment two so there's a lot of grading being done this week. The TA's are pretty busy.
Also a reminder for you guys, hopefully you've been working hard on your projects now that most of you
are done with the midterm so your project milestones will be due on Tuesday so any sort of last minute changes that you had in your projects,
I know some people decided to switch projects after the proposal, some teams reshuffled a little bit,
that's fine but your milestone should reflect the project that you're actually doing for the rest of the quarter. So hopefully that's going out well.
I know there's been a lot of worry and stress on Piazza, wondering about assignment three.
So we're working on that as hard as we can but that's actually a bit of a new assignment, it's changing a bit from last year
so it will be out as soon as possible, hopefully today or tomorrow.
Although we promise that whenever it comes out you'll have two weeks to finish it so try not to stress out about that too much.
But I'm pretty excited, I think assignment three will be really cool, has a lot of cool,
it'll cover a lot of really cool material.
So another thing, last time in lecture we mentioned this thing called the Train Game
which is this really cool thing we've been working on sort of as a side project a little bit.
So this is an interactive tool that you guys can go on and use to explore a little bit the process
of tuning hyperparameters in practice so we hope that,
so this is again totally not required for the course. Totally optional, but if you do we will offer
a small amount of extra credit for those of you
who want to do well and participate on this.
And we'll send out exactly some more details later this afternoon on Piazza.
But just a bit of a demo for what exactly is this thing. So you'll get to go in and we've changed the name
from Train Game to HyperQuest because you're questing
to solve, to find the best hyperparameters for your model
so this is really cool, it'll be an interactive tool that you can use to explore the training of hyperparameters
interactively in your browser.
So you'll login with your student ID and name.
You'll fill out a little survey with some of your experience on deep learning
then you'll read some instructions. So in this game you'll be shown some random data set
on every trial.
This data set might be images or it might be vectors and your goal is to train a model by picking
the right hyperparameters interactively to perform as well as you can on the validation set
of this random data set.
And it'll sort of keep track of your performance over time and there'll be a leaderboard,
it'll be really cool.
So every time you play the game, you'll get some statistics about your data set.
In this case we're doing a classification problem with 10 classes.
You can see down at the bottom you have these statistics about random data set, we have 10 classes.
The input data size is three by 32 by 32 so this is some image data set and we can see
that in this case we have 8500 examples in the training set and 1500 examples in the validation set.
These are all random, they'll change a little bit every time.
Based on these data set statistics you'll make some choices on your initial learning rate, your initial network size,
and your initial dropout rate.
Then you'll see a screen like this where it'll run one epoch with those chosen hyperparameters,
show you on the right here you'll see two plots. One is your training and validation loss
for that first epoch.
Then you'll see your training and validation accuracy
for that first epoch and based on the gaps that you see in these two graphs you can make choices interactively to change the learning rates and hyperparameters
for the next epoch.
So then you can either choose to continue training with the current or changed hyperparameters,
you can also stop training, or you can revert to go back to the previous checkpoint
in case things got really messed up.
So then you'll get to make some choice, so here we'll decide to continue training
and in this case you could go and set new learning rates
and new hyperparameters for the next epoch of training.
You can also, kind of interesting here, you can actually grow the network interactively
during training in this demo.
There's this cool trick from a couple recent papers where you can either take existing layers
and make them wider or add new layers to the network in the middle of training while still maintaining
the same function in the network so you can do that
to increase the size of your network in the middle of training here which is kind of cool.
So then you'll make choices over several epochs and eventually your final validation accuracy
will be recorded and we'll have some leaderboard
that compares your score on that data set
to some simple baseline models.
And depending on how well you do on this leaderboard we'll again offer some small amounts of extra credit
for those of you who choose to participate.
So this is again, totally optional, but I think
it can be a really cool learning experience for you guys to play around with and explore how hyperparameters
affect the learning process.
Also, it's really useful for us. You'll help science out by participating in this experiment.
We're pretty interested in seeing how people behave when they train neural networks so you'll be helping us out
as well if you decide to play this.
But again, totally optional, up to you.
Any questions on that?
Hopefully at some point but it's. So the question was will this be a paper
or whatever eventually?
Hopefully but it's really early stages of this project so I can't make any promises but I hope so.
But I think it'll be really cool.
[laughing]
Yeah, so the question is how can you add layers during training?
I don't really want to get into that right now but the paper to read is Net2Net by Ian Goodfellow's
one of the authors and there's another paper
from Microsoft called Network Morphism.
So if you read those two papers you can see how this works.
Okay, so last time, a bit of a reminder before we had the midterm last time we talked
about recurrent neural networks.
We saw that recurrent neural networks can be used for different types of problems.
In addition to one to one we can do one to many, many to one, many to many.
We saw how this can apply to language modeling
and we saw some cool examples of applying neural networks to model different sorts of languages at the character level
and we sampled these artificial math and Shakespeare and C source code.
We also saw how similar things could be applied to image captioning by connecting a CNN feature extractor
together with an RNN language model.
And we saw some really cool examples of that.
We also talked about the different types of RNN's. We talked about this Vanilla RNN.
I also want to mention that this is sometimes called a Simple RNN or an Elman RNN so you'll see
all of these different terms in literature.
We also talked about the Long Short Term Memory or LSTM.
And we talked about how the gradient, the LSTM has this crazy set of equations
but it makes sense because it helps improve gradient flow
during back propagation and helps this thing model
more longer term dependencies in our sequences.
So today we're going to switch gears and talk about a whole bunch of different exciting tasks.
We're going to talk about, so so far we've been talking about mostly the image classification problem.
Today we're going to talk about various types of other computer vision tasks where you actually want to go in
and say things about the spatial pixels inside your images so we'll see segmentation, localization, detection,
a couple other different computer vision tasks
and how you can approach these with convolutional neural networks.
So as a bit of refresher, so far the main thing we've been talking about in this class
is image classification so here we're going to have
some input image come in. That input image will go through
some deep convolutional network,
that network will give us some feature vector of maybe 4096 dimensions in the case of AlexNet RGB
and then from that final feature vector we'll have some fully-connected,
some final fully-connected layer
that gives us 1000 numbers for the different class scores
that we care about where 1000 is maybe the number of classes in ImageNet in this example.
And then at the end of the day what the network does is we input an image
and then we output a single category label
saying what is the content of this entire image as a whole.
But this is maybe the most basic possible task in computer vision and there's a whole bunch
of other interesting types of tasks
that we might want to solve using deep learning.
So today we're going to talk about several of these different tasks and step through each of these
and see how they all work with deep learning.
So we'll talk about these more in detail about what each problem is as we get to it
but this is kind of a summary slide
that we'll talk first about semantic segmentation.
We'll talk about classification and localization, then we'll talk about object detection,
and finally a couple brief words about instance segmentation.
So first is the problem of semantic segmentation.
In the problem of semantic segmentation, we want to input an image and then output a decision
of a category for every pixel in that image
so for every pixel in this, so this input image for example is this cat walking through the field, he's very cute.
And in the output we want to say for every pixel is that pixel a cat or grass or sky or trees
or background or some other set of categories.
So we're going to have some set of categories just like we did in the image classification case
but now rather than assigning a single category labeled to the entire image, we want to produce
a category label for each pixel of the input image.
And this is called semantic segmentation.
So one interesting thing about semantic segmentation is that it does not differentiate instances
so in this example on the right we have this image with two cows where they're standing right next
to each other and when we're talking about semantic segmentation we're just labeling all the pixels
independently for what is the category of that pixel.
So in the case like this where we have two cows right next to each other the output does not make
any distinguishing, does not distinguish
between these two cows.
Instead we just get a whole mass of pixels that are all labeled as cow.
So this is a bit of a shortcoming of semantic segmentation and we'll see how we can fix this later
when we move to instance segmentation.
But at least for now we'll just talk about semantic segmentation first.
So you can imagine maybe using a class, so one potential approach for attacking
semantic segmentation might be through classification.
So there's this, you could use this idea of a sliding window approach to semantic segmentation.
So you might imagine that we take our input image and we break it up into many many small, tiny local crops
of the image so in this example we've taken
maybe three crops from around the head of this cow
and then you could imagine taking each of those crops and now treating this as a classification problem.
Saying for this crop, what is the category of the central pixel of the crop?
And then we could use all the same machinery that we've developed for classifying entire images
but now just apply it on crops rather than
on the entire image.
And this would probably work to some extent but it's probably not a very good idea.
So this would end up being super super computationally expensive because we want to label
every pixel in the image, we would need a separate crop for every pixel in that image and this would be
super super expensive to run forward and backward
passes through.
And moreover, we're actually, if you think about this we can actually share computation between different
patches so if you're trying to classify two patches
that are right next to each other and actually overlap
then the convolutional features of those patches
will end up going through the same convolutional layers and we can actually share a lot of the computation
when applying this to separate passes
or when applying this type of approach
to separate patches in the image.
So this is actually a terrible idea and nobody does this and you should probably not do this
but it's at least the first thing you might think of if you were trying to think about semantic segmentation.
Then the next idea that works a bit better is this idea of a fully convolutional network right.
So rather than extracting individual patches from the image and classifying these patches independently,
we can imagine just having our network be a whole giant stack of convolutional layers with no fully connected
layers or anything so in this case we just have a bunch
of convolutional layers that are all maybe three by three with zero padding or something like that
so that each convolutional layer preserves the spatial size
of the input and now if we pass our image
through a whole stack of these convolutional layers,
then the final convolutional layer could just output a tensor of something by C by H by W
where C is the number of categories that we care about
and you could see this tensor as just giving our classification scores for every pixel
in the input image at every location in the input image.
And we could compute this all at once with just some giant stack of convolutional layers.
And then you could imagine training this thing by putting a classification loss at every pixel
of this output, taking an average over those pixels
in space, and just training this kind of network through normal, regular back propagation.
Question?
Oh, the question is how do you develop training data for this?
It's very expensive right. So the training data for this would be
we need to label every pixel in those input images
so there's tools that people sometimes have online where you can go in and sort of draw contours
around the objects and then fill in regions
but in general getting this kind of training data is very expensive.
Yeah, the question is what is the loss function?
So here since we're making a classification decision per pixel then we put a cross entropy loss
on every pixel of the output.
So we have the ground truth category label for every pixel in the output,
then we compute across entropy loss between every pixel in the output
and the ground truth pixels and then
take either a sum or an average over space and then sum or average over the mini-batch.
Question?
Yeah, yeah. Yeah, the question is do we assume
that we know the categories?
So yes, we do assume that we know the categories up front
so this is just like the image classification case.
So an image classification we know at the start of training based on our data set that maybe there's 10 or 20
or 100 or 1000 classes that we care about
for this data set and then here we are fixed to that set of classes that are fixed for the data set.
So this model is relatively simple and you can imagine this working reasonably well
assuming that you tuned all the hyperparameters right
but it's kind of a problem right.
So in this setup, since we're applying a bunch of convolutions that are all keeping the same
spatial size of the input image,
this would be super super expensive right.
If you wanted to do convolutions that maybe have 64 or 128 or 256 channels for those convolutional filters
which is pretty common in a lot of these networks,
then running those convolutions on this high resolution input image over a sequence of layers would be
extremely computationally expensive
and would take a ton of memory.
So in practice, you don't usually see networks with this architecture.
Instead you tend to see networks that look something like this where we have some downsampling
and then some upsampling of the feature map
inside the image.
So rather than doing all the convolutions of the full spatial resolution of the image,
we'll maybe go through a small number of convolutional layers at the original resolution
then downsample that feature map using something like max pooling or strided convolutions
and sort of downsample, downsample,
so we have convolutions in downsampling and convolutions in downsampling
that look much like a lot of the classification networks that you see but now the difference is that
rather than transitioning to a fully connected layer like you might do in an image classification setup,
instead we want to increase the spatial resolution
of our predictions in the second half of the network
so that our output image can now be the same size as our input image and this ends up being
much more computationally efficient
because you can make the network very deep and work at a lower spatial resolution
for many of the layers at the inside of the network.
So we've already seen examples of downsampling when it comes to convolutional networks.
We've seen that you can do strided convolutions or various types of pooling to reduce the spatial size
of the image inside a network but we haven't
really talked about upsampling and the question
you might be wondering is what are these upsampling layers actually look like inside the network?
And what are our strategies for increasing the size of a feature map inside the network?
Sorry, was there a question in the back?
Yeah, so the question is how do we upsample?
And the answer is that's the topic of the next couple slides.
[laughing]
So one strategy for upsampling is something like unpooling so we have this notion of pooling
to downsample so we talked about average pooling
or max pooling so when we talked about average pooling
we're kind of taking a spatial average within a receptive field of each pooling region.
One kind of analog for upsampling is this idea of nearest neighbor unpooling.
So here on the left we see this example of nearest neighbor unpooling where our input
is maybe some two by two grid and our output
is a four by four grid and now in our output
we've done a two by two stride two nearest neighbor unpooling or upsampling where we've just duplicated
that element for every point in our two by two
receptive field of the unpooling region.
Another thing you might see is this bed of nails unpooling or bed of nails upsampling where you'll just take,
again we have a two by two receptive field for our unpooling regions and then you'll take the,
in this case you make it all zeros except for one element of the unpooling region so in this case we've taken all of our inputs and always put them in the upper left hand corner of this unpooling region and everything else is zeros.
And this is kind of like a bed of nails
because the zeros are very flat, then you've got these things poking up for the values at these various non-zero regions.
Another thing that you see sometimes which was alluded to by the question a minute ago is this idea of max unpooling
so in a lot of these networks they tend to be symmetrical where we have a downsampling portion of the network and then an upsampling portion of the network with a symmetry between those two portions of the network.
So sometimes what you'll see is this idea of max unpooling where for each unpooling, for each upsampling layer, it is associated with one of the pooling layers in the first half of the network and now in the first half,
in the downsampling when we do max pooling we'll actually remember which element of the receptive field during max pooling was used to do the max pooling
and now when we go through the rest of the network then we'll do something that looks like this bed of nails upsampling except rather than always putting the elements in the same position, instead we'll stick it
into the position that was used in the corresponding max pooling step earlier in the network.
I'm not sure if that explanation was clear but hopefully the picture makes sense.
Yeah, so then you just end up filling the rest with zeros.
So then you fill the rest with zeros and then you stick the elements from the low resolution patch up into the high resolution patch
at the points where the max pooling took place at the corresponding max pooling there.
Okay, so that's kind of an interesting idea.
Sorry, question?
Oh yeah, so the question is why is this a good idea? Why might this matter?
So the idea is that when we're doing semantic segmentation we want our predictions to be pixel perfect right.
We kind of want to get those sharp boundaries and those tiny details in our predictive segmentation
so now if you're doing this max pooling, there's this sort of heterogeneity that's happening inside the feature map due to the max pooling
where from the low resolution image you don't know, you're sort of losing spatial information in some sense by you don't know where that feature vector came from in the local receptive field after max pooling.
So if you actually unpool by putting the vector in the same slot you might think that that might help us handle these fine details a little bit better
and help us preserve some of that spatial information that was lost during max pooling.
Question?
The question is does this make things easier for back prop?
Yeah, I guess, I don't think it changes the back prop dynamics too much because storing these indices is not a huge computational overhead.
They're pretty small in comparison to everything else.
So another thing that you'll see sometimes is this idea of transpose convolution.
So transpose convolution, so for these various types of unpooling that we just talked about,
these bed of nails, this nearest neighbor, this max unpooling, all of these are kind of
a fixed function, they're not really learning exactly how to do the upsampling so if you think about something
like strided convolution, strided convolution
is kind of like a learnable layer that learns the way that the network wants to perform downsampling at that layer.
And by analogy with that there's this type of layer called a transpose convolution that lets us do kind of learnable upsampling.
So it will both upsample the feature map and learn some weights about how it wants to do that upsampling.
And this is really just another type of convolution so to see how this works remember how a normal
three by three stride one pad one convolution would work.
That for this kind of normal convolution that we've seen many times now in this class,
our input might by four by four, our output might be four by four,
and now we'll have this three by three kernel and we'll take an inner product between, we'll plop down that kernel at the corner of the image,
take an inner product, and that inner product will give us the value and the activation in the upper left hand corner of our output.
And we'll repeat this for every receptive field in the image.
Now if we talk about strided convolution then strided convolution ends up looking pretty similar.
However, our input is maybe a four by four region and our output is a two by two region.
But we still have this idea of taking, of there being some three by three filter or kernel that we plop down in the corner of the image, take an inner product and use that to compute a value of the activation and the output.
But now with strided convolution the idea is that we're moving that, rather than popping down that filter at every possible point in the input,
instead we're going to move the filter by two pixels in the input every time we move the filter by one pixel, every time we move by one pixel in the output.
Right so this stride of two gives us a ratio between how much do we move in the input versus how much do we move in the output.
So when you do a strided convolution with stride two this ends up downsampling the image or the feature map by a factor of two in kind of a learnable way.
And now a transpose convolution is sort of the opposite in a way so here our input will be a two by two region and our output will be a four by four region.
But now the operation that we perform with transpose convolution is a little bit different.
Now so rather than taking an inner product instead what we're going to do is we're going to take the value of our input feature map
at that upper left hand corner and that'll be some scalar value in the upper left hand corner.
We're going to multiply the filter by that scalar value and then copy those values over to this three by three
region in the output so rather than taking an inner product with our filter and the input, instead our input
gives weights that we will use to weight the filter and then our output will be weighted copies of the filter that are weighted by the values in the input.
And now we can do this sort of same ratio trick in order to upsample so now when we move one pixel in the input now we can plop our filter down two pixels away in the output and it's the same trick
that now the blue pixel in the input is some scalar value and we'll take that scalar value, multiply it by the values in the filter,
and copy those weighted filter values into this new region in the output.
The tricky part is that sometimes these receptive fields in the output can overlap now and now when these
receptive fields in the output overlap we just sum the results in the output.
So then you can imagine repeating this everywhere and repeating this process everywhere and this ends up doing sort of a learnable upsampling
where we use these learned convolutional filter weights to upsample the image and increase the spatial size.
By the way, you'll see this operation go by a lot of different names in literature.
Sometimes this gets called things like deconvolution
which I think is kind of a bad name but you'll see it
out there in papers so from a signal processing perspective deconvolution means the inverse operation to convolution
which this is not however you'll frequently see this type of layer called a deconvolution layer
in some deep learning papers so be aware of that, watch out for that terminology.
You'll also sometimes see this called upconvolution which is kind of a cute name.
Sometimes it gets called fractionally strided convolution
because if we think of the stride as the ratio in step between the input and the output then now this is something like a stride one half convolution because of this ratio
of one to two between steps in the input and steps in the output.
This also sometimes gets called a backwards strided convolution because if you think about it
and work through the math this ends up being the same, the forward pass of a transpose convolution
ends up being the same mathematical operation as the backwards pass in a normal convolution
so you might have to take my word for it, that might not be super obvious when you first look at this but that's kind of a neat fact so you'll sometimes see that name as well.
And as maybe a bit of a more concrete example of what this looks like I think it's maybe a little easier to see in one dimension so if we imagine,
so here we're doing a three by three transpose convolution in one dimension.
Sorry, not three by three, a three by one transpose convolution in one dimension.
So our filter here is just three numbers. Our input is two numbers and now you can see
that in our output we've taken the values in the input, used them to weight the values of the filter and plopped down those weighted filters in the output
with a stride of two and now where these receptive fields overlap in the output then we sum.
So you might be wondering, this is kind of a funny name. Where does the name transpose convolution come from and why is that actually my preferred name for this operation?
So that comes from this kind of neat interpretation of convolution.
So it turns out that any time you do convolution you can always write convolution as a matrix multiplication.
So again, this is kind of easier to see with a one-dimensional example
but here we've got some weight. So we're doing a one-dimensional convolution of a weight vector x which has three elements,
and an input vector, a vector, which has four elements, A, B, C, D.
So here we're doing a three by one convolution with stride one and you can see that we can frame this whole operation as a matrix multiplication
where we take our convolutional kernel x and turn it into some matrix capital X
which contains copies of that convolutional kernel that are offset by different regions.
And now we can take this giant weight matrix X and do a matrix vector multiplication between x and our input a and this just produces the same result as convolution.
And now with transpose convolution means that we're going to take this same weight matrix but now we're going to multiply by the transpose of that same weight matrix.
So here you can see the same example for this stride one convolution on the left and the corresponding stride one transpose convolution on the right.
And if you work through the details you'll see that when it comes to stride one,
a stride one transpose convolution also ends up being a stride one normal convolution so there's a little bit
of details in the way that the border and the padding are handled but it's fundamentally the same operation.
But now things look different when you talk about a stride of two.
So again, here on the left we can take a stride two convolution and write out this stride two convolution as a matrix multiplication.
And now the corresponding transpose convolution is no longer a convolution so if you look
through this weight matrix and think about how convolutions end up getting represented in this way
then now this transposed matrix for the stride two convolution is something fundamentally different from the original normal convolution operation
so that's kind of the reasoning behind the name and that's why I think that's kind of the nicest name to call this operation by.
Sorry, was there a question?
Sorry?
It's very possible there's a typo in the slide so please point out on Piazza and I'll fix it but I hope the idea was clear.
Is there another question? Okay, thank you [laughing]. Yeah, so, oh no lots of questions.
Yeah, so the issue is why do we sum and not average?
So the reason we sum is due to this transpose convolution formula zone so that's the reason why we sum
but you're right that you actually, this is kind of a problem that the magnitudes will actually vary in the output depending on how many receptive fields were in the output.
So actually in practice this is something that people started to point out very recently and somewhat
switched away from this stride, so using three by three stride two transpose convolution upsampling can sometimes produce some checkerboard artifacts in the output exactly due to that problem.
So what I've seen in a couple more recent papers is maybe to use four by four stride two or two by two stride two transpose convolution for upsampling and that helps alleviate that problem a little bit.
- Yeah, so the question is what is a stride half convolution - and where does that terminology come from?
I think that was from my paper. So that was actually, yes that was definitely this.
So at the time I was writing that paper I was kind of into the name fractionally strided convolution
but after thinking about it a bit more I think transpose convolution is probably the right name.
So then this idea of semantic segmentation actually ends up being pretty natural.
You just have this giant convolutional network with downsampling and upsampling inside the network
and now our downsampling will be by strided convolution
or pooling, our upsampling will be by transpose convolution or various types of unpooling or upsampling
and we can train this whole thing end to end with back propagation using this cross entropy loss over every pixel.
So this is actually pretty cool that we can take a lot of the same machinery that we already learned for image classification and now just apply it
very easily to extend to new types of problems so that's super cool.
So the next task that I want to talk about is this idea of classification plus localization.
So we've talked about image classification a lot
where we want to just assign a category label to the input image but sometimes you might want to know a little bit more about the image.
In addition to predicting what the category is, in this case the cat, you might also want to know where is that object in the image?
So in addition to predicting the category label cat, you might also want to draw a bounding box around the region of the cat in that image.
And classification plus localization, the distinction here between this and object detection
is that in the localization scenario you assume ahead of time that you know there's exactly one object in the image that you're looking for or maybe more than one
but you know ahead of time that we're going to make some classification decision about this image and we're going to produce exactly one bounding box that's going to tell us where that object is located
in the image so we sometimes call that task classification plus localization.
And again, we can reuse a lot of the same machinery that we've already learned from image classification in order to tackle this problem.
So kind of a basic architecture for this problem looks something like this.
So again, we have our input image, we feed our input image through some giant convolutional network, this is Alex, this is AlexNet for example, which will give us some final vector summarizing the content of the image.
Then just like before we'll have some fully connected layer that goes from that final vector to our class scores.
But now we'll also have another fully connected layer that goes from that vector to four numbers.
Where the four numbers are something like the height, the width, and the x and y positions of that bounding box.
And now our network will produce these two different outputs, one is this set of class scores,
and the other are these four numbers giving the coordinates of the bounding box in the input image.
And now during training time, when we train this network we'll actually have two losses so in this scenario
we're sort of assuming a fully supervised setting
so we assume that each of our training images is annotated with both a category label and also a ground truth bounding box for that category in the image.
So now we have two loss functions.
We have our favorite softmax loss that we compute using the ground truth category label and the predicted class scores,
and we also have some kind of loss that gives us some measure of dissimilarity between our predicted coordinates for the bounding box and our actual coordinates for the bounding box.
So one very simple thing is to just take an L2 loss between those two and that's kind of the simplest thing that you'll see in practice although sometimes
people play around with this and maybe use L1 or smooth L1 or they parametrize the bounding box a little bit differently but the idea is always the same,
that you have some regression loss between your predicted bounding box coordinates and the ground truth bounding box coordinates.
Question? Sorry, go ahead.
So the question is, is this a good idea to do all at the same time?
Like what happens if you misclassify, should you even look at the box coordinates?
So sometimes people get fancy with it, so in general it works okay.
It's not a big problem, you can actually train a network to do both of these things at the same time
and it'll figure it out but sometimes things can get tricky in terms of misclassification so sometimes what you'll see
for example is that rather than predicting a single box you might make predictions like a separate prediction of the box for each category and then only apply loss
to the predicted box corresponding to the ground truth category.
So people do get a little bit fancy with these things that sometimes helps a bit in practice.
But at least this basic setup, it might not be perfect or it might not be optimal but it will work and it will do something.
Was there a question in the back?
Yeah, so that's the question is do these losses have different units, do they dominate the gradient?
So this is what we call a multi-task loss
so whenever we're taking derivatives we always want to take derivative of a scalar with respect to our network parameters and use that derivative to take gradient steps.
But now we've got two scalars that we want to both minimize
so what you tend to do in practice is have some additional hyperparameter that gives you some weighting between these two losses so you'll take a weighted sum of these two different loss functions to give our final scalar loss.
And then you'll take your gradients with respect to this weighted sum of the two losses.
And this ends up being really really tricky because this weighting parameter is a hyperparameter that you need to set but it's kind of different
from some of the other hyperparameters that we've seen so far in the past right
because this weighting hyperparameter actually changes the value of the loss function
so one thing that you might often look at when you're trying to set hyperparameters is you might make different hyperparameter choices and see what happens to the loss under different choices of hyperparameters.
But in this case because the loss actually, because the hyperparameter affects the absolute value of the loss making those comparisons becomes kind of tricky.
So setting that hyperparameter is somewhat difficult.
And in practice, you kind of need to take it on a case by case basis for exactly the problem you're solving but my general strategy for this
is to have some other metric of performance that you care about other than the actual loss value
which then you actually use that final performance metric to make your cross validation choices rather than looking at the value of the loss to make those choices.
Question?
So the question is why do we do this all at once? Why not do this separately?
Yeah, so the question is why don't we fix the big network and then just only learn separate fully connected layers for these two tasks?
People do do that sometimes and in fact that's probably the first thing you should try if you're faced with a situation like this but in general
whenever you're doing transfer learning you always get better performance if you fine tune the whole system jointly because there's probably some mismatch between the features,
if you train on ImageNet and then you use that network for your data set you're going to get better performance on your data set if you can also change the network.
But one trick you might see in practice sometimes is that you might freeze that network then train those two things separately until convergence
and then after they converge then you go back and jointly fine tune the whole system.
So that's a trick that sometimes people do in practice in that situation.
And as I've kind of alluded to this big network is often a pre-trained network that is taken from ImageNet for example.
So a bit of an aside, this idea of predicting some fixed number of positions in the image
can be applied to a lot of different problems beyond just classification plus localization.
One kind of cool example is human pose estimation.
So here we want to take an input image is a picture of a person.
We want to output the positions of the joints for that person and this actually allows the network to predict what is the pose of the human.
Where are his arms, where are his legs, stuff like that,
and generally most people have the same number of joints. That's a bit of a simplifying assumption,
it might not always be true but it works for the network.
So for example one parameterization that you might see
in some data sets is define a person's pose
by 14 joint positions.
Their feet and their knees and their hips
and something like that and now when we train the network
then we're going to input this image of a person
and now we're going to output 14 numbers in this case
giving the x and y coordinates for each of those 14 joints.
And then you apply some kind of regression loss
on each of those 14 different predicted points
and just train this network with back propagation again.
Yeah, so you might see an L2 loss but people play around
with other regression losses here as well.
Question?
So the question is what do I mean
when I say regression loss?
So I mean something other than cross entropy
or softmax right.
When I say regression loss I usually mean
like an L2 Euclidean loss or an L1 loss
or sometimes a smooth L1 loss.
But in general classification versus regression
is whether your output is categorical or continuous
so if you're expecting a categorical output
like you ultimately want to make a classification decision
over some fixed number of categories
then you'll think about a cross entropy loss,
softmax loss or these SVM margin type losses
that we talked about already in the class.
But if your expected output is to be some continuous value,
in this case the position of these points,
then your output is continuous so you tend to use
different types of losses in those situations.
Typically an L2, L1, different kinds of things there.
So sorry for not clarifying that earlier.
But the bigger point here is that for any time
you know that you want to make some fixed number
of outputs from your network, if you know for example.
Maybe you knew that you wanted to,
you knew that you always are going to have pictures
of a cat and a dog and you want to predict both
the bounding box of the cat and the bounding box of the dog
in that case you'd know that you have a fixed number
of outputs for each input so you might imagine
hooking up this type of regression
classification plus localization framework
for that problem as well.
So this idea of some fixed number of regression outputs
can be applied to a lot of different problems
including pose estimation.
So the next task that I want to talk about is object detection
and this is a really meaty topic.
This is kind of a core problem in computer vision
and you could probably teach a whole seminar class
on just the history of object detection
and various techniques applied there.
So I'll be relatively brief and try to go over
the main big ideas of object detection plus deep learning
that have been used in the last couple of years.
But the idea in object detection is that
we again start with some fixed set of categories
that we care about, maybe cats and dogs and fish or whatever
but some fixed set of categories that we're interested in.
And now our task is that given our input image,
every time one of those categories appears in the image,
we want to draw a box around it and we want to predict
the category of that box so this is different
from classification plus localization
because there might be a varying number of outputs
for every input image.
You don't know ahead of time how many objects you expect
to find in each image so that's,
this ends up being a pretty challenging problem.
So we've seen graphs, so this is kind of interesting.
We've seen this graph many times of the ImageNet
classification performance as a function of years
and we saw that it just got better and better every year
and there's been a similar trend with object detection
because object detection has again been one
of these core problems in computer vision
that people have cared about for a very long time.
So this slide is due to Ross Girshick
who's worked on this problem a lot and it shows
the progression of object detection performance
on this one particular data set called PASCAL VOC
which has been relatively used for a long time
in the object detection community.
And you can see that up until about 2012
performance on object detection started to stagnate
and slow down a little bit and then in 2013
was when some of the first deep learning approaches
to object detection came around and you could see
that performance just shot up very quickly
getting better and better year over year.
One thing you might notice is that this plot ends in 2015 and it's actually continued to go up since then
so the current state of the art in this data set is well over 80 and in fact a lot of recent papers don't even report results on this data set anymore because it's considered too easy.
So it's a little bit hard to know, I'm not actually sure what is the state of the art number on this data set but it's off the top of this plot.
Sorry, did you have a question? Nevermind.
Okay, so as I already said this is different from localization because there might be differing numbers of objects for each image.
So for example in this cat on the upper left there's only one object so we only need to predict four numbers but now for this image in the middle
there's three animals there so we need our network to predict 12 numbers, four coordinates for each bounding box.
Or in this example of many many ducks then you want your network to predict a whole bunch of numbers. Again, four numbers for each duck.
So this is quite different from object detection. Sorry object detection is quite different from localization
because in object detection you might have varying numbers of objects in the image and you don't know ahead of time how many you expect to find.
So as a result, it's kind of tricky if you want to think of object detection as a regression problem.
So instead, people tend to work, use kind of a different paradigm when thinking about object detection.
So one approach that's very common and has been used for a long time in computer vision is this idea of sliding window approaches to object detection.
So this is kind of similar to this idea of taking small patches and applying that for semantic segmentation and we can apply a similar idea for object detection.
So the ideas is that we'll take different crops from the input image, in this case we've got this crop
in the lower left hand corner of our image and now we take that crop, feed it through our convolutional network
and our convolutional network does a classification decision on that input crop.
It'll say that there's no dog here, there's no cat here,
and then in addition to the categories that we care about we'll add an additional category called background
and now our network can predict background in case it doesn't see any of the categories that we care about, so then when we take this crop
from the lower left hand corner here then our network would hopefully predict background and say that no, there's no object here.
Now if we take a different crop then our network would predict dog yes, cat no, background no.
We take a different crop we get dog yes, cat no, background no.
Or a different crop, dog no, cat yes, background no. Does anyone see a problem here?
Yeah, the question is how do you choose the crops? So this is a huge problem right.
Because there could be any number of objects in this image, these objects could appear at any location in the image,
these objects could appear at any size in the image, these objects could also appear at any aspect ratio
in the image, so if you want to do kind of a brute force sliding window approach you'd end up having to test thousands, tens of thousands, many many many many different crops in order to tackle this problem with a brute force sliding window approach.
And in the case where every one of those crops is going to be fed through a giant convolutional network, this would be completely computationally intractable.
So in practice people don't ever do this sort of brute force sliding window approach for object detection using convolutional networks.
Instead there's this cool line of work called region proposals that comes from, this is not using deep learning typically.
These are slightly more traditional computer vision
techniques but the idea is that a region proposal network kind of uses more traditional signal processing, image processing type things to make some list
of proposals for where, so given an input image, a region proposal network will then give you something like a thousand boxes where an object might be present.
So you can imagine that maybe we do some local, we look for edges in the image and try to draw boxes that contain closed edges or something like that.
These various types of image processing approaches, but these region proposal networks will basically look for blobby regions in our input image and then give us
some set of candidate proposal regions where objects might be potentially found. And these are relatively fast-ish to run
so one common example of a region proposal method that you might see is something called Selective Search
which I think actually gives you 2000 region proposals, not the 1000 that it says on the slide.
So you kind of run this thing and then after about two seconds of turning on your CPU it'll spit out 2000 region proposals in the input image where objects are likely to be found
so there'll be a lot of noise in those. Most of them will not be true objects but there's a pretty high recall.
If there is an object in the image then it does tend to get covered by these region proposals from Selective Search.
So now rather than applying our classification network to every possible location and scale in the image
instead what we can do is first apply one of these region proposal networks to get some set of proposal regions where objects are likely located
and now apply a convolutional network for classification to each of these proposal regions and this will end up being much more computationally tractable
than trying to do all possible locations and scales.
And this idea all came together in this paper called R-CNN from a few years ago that does exactly that.
So given our input image in this case we'll run some region proposal network to get our proposals, these are also sometimes called
regions of interest or ROI's so again Selective Search
gives you something like 2000 regions of interest.
Now one of the problems here is that these input, these regions in the input image could have different sizes
but if we're going to run them all through a convolutional network our classification, our convolutional networks for classification
all want images of the same input size typically due to the fully connected net layers and whatnot
so we need to take each of these region proposals and warp them to that fixed square size that is expected as input to our downstream network.
So we'll crop out those region proposal, those regions corresponding to the region proposals, we'll warp them to that fixed size,
and then we'll run each of them through a convolutional network
which will then use in this case an SVM to make a classification decision for each of those, to predict categories for each of those crops.
And then I lost a slide.
But it'll also, not shown in the slide right now but in addition R-CNN also predicts a regression, like a correction to the bounding box in addition for each of these input region proposals
because the problem is that your input region proposals are kind of generally in the right position for an object but they might not be perfect so in addition R-CNN will,
in addition to category labels for each of these proposals, it'll also predict four numbers that are kind of an offset or a correction to the box that was predicted at the region proposal stage.
So then again, this is a multi-task loss and you would train this whole thing.
Sorry was there a question?
The question is how much does the change in aspect ratio impact accuracy?
It's a little bit hard to say.
I think there's some controlled experiments in some of these papers but I'm not sure
I can give a generic answer to that.
Question?
The question is is it necessary for regions of interest to be rectangles?
So they typically are because it's tough to warp these non-region things but once you move
to something like instant segmentation then you sometimes get proposals that are not rectangles.
If you actually do care about predicting things that are not rectangles.
Is there another question?
Yeah, so the question is are the region proposals learned so in R-CNN it's a traditional thing.
These are not learned, this is kind of some fixed algorithm that someone wrote down but we'll see in a couple minutes
that we can actually, we've changed that a little bit in the last couple of years.
Is there another question?
The question is is the offset always inside the region of interest?
The answer is no, it doesn't have to be.
You might imagine that suppose the region of interest put a box around a person but missed the head then you could imagine the network inferring
that oh this is a person but people usually have heads so the network showed the box should be a little bit higher.
So sometimes the final predicted boxes will be outside the region of interest.
Question?
Yeah. Yeah the question is you have a lot of ROI's that don't correspond to true objects?
And like we said, in addition to the classes that you actually care about you add an additional background class so your class scores can also
predict background to say that there was no object here.
Question?
Yeah, so the question is what kind of data do we need
and yeah, this is fully supervised in the sense that our training data has each image, consists of images. Each image has all the object categories marked with bounding boxes for each instance of that category.
There are definitely papers that try to approach this like oh what if you don't have the data. What if you only have that data for some images? Or what if that data is noisy but at least
in the generic case you assume full supervision of all objects in the images at training time.
Okay, so I think we've kind of alluded to this but there's kind of a lot of problems with this R-CNN framework.
And actually if you look at the figure here on the right you can see that additional bounding box head so I'll put it back.
But this is kind of still computationally pretty expensive
because if we've got 2000 region proposals, we're running each of those proposals independently, that can be pretty expensive.
There's also this question of relying on this fixed region proposal network, this fixed region proposals, we're not learning them so that's kind of a problem.
And just in practice it ends up being pretty slow
so in the original implementation R-CNN would actually dump all the features to disk so it'd take hundreds of gigabytes of disk space to store all these features.
Then training would be super slow since you have to make all these different forward and backward passes
through the image and it took something like 84 hours is one number they've recorded for training time so this is super super slow.
And now at test time it's also super slow, something like roughly 30 seconds minute per image
because you need to run thousands of forward passes through the convolutional network for each of these region proposals so this ends up being pretty slow.
Thankfully we have fast R-CNN that fixed a lot of these problems so when we do fast R-CNN then it's going to look kind of the same.
We're going to start with our input image but now rather than processing each region of interest separately instead we're going to run the entire image
through some convolutional layers all at once to give this high resolution convolutional feature map corresponding to the entire image.
And now we still are using some region proposals from some fixed thing like Selective Search
but rather than cropping out the pixels of the image corresponding to the region proposals,
instead we imagine projecting those region proposals onto this convolutional feature map and then taking crops from the convolutional feature map corresponding to each proposal rather than taking crops directly from the image.
And this allows us to reuse a lot of this expensive convolutional computation across the entire image when we have many many crops per image.
But again, if we have some fully connected layers downstream those fully connected layers are expecting some fixed-size input
so now we need to do some reshaping of those crops from the convolutional feature map
and they do that in a differentiable way using something they call an ROI pooling layer.
Once you have these warped crops from the convolutional feature map then you can run these things through some
fully connected layers and predict your classification scores and your linear regression offsets to the bounding boxes.
And now when we train this thing then we again have a multi-task loss that trades off between these two constraints and during back propagation
we can back prop through this entire thing and learn it all jointly.
This ROI pooling, it looks kind of like max pooling. I don't really want to get into the details of that right now.
And in terms of speed if we look at R-CNN versus fast R-CNN versus this other model called SPP net which is kind of in between the two,
then you can see that at training time fast R-CNN is something like 10 times faster to train
because we're sharing all this computation between different feature maps.
And now at test time fast R-CNN is super fast
and in fact fast R-CNN is so fast at test time that its computation time is actually dominated by computing region proposals.
So we said that computing these 2000 region proposals using Selective Search takes something like two seconds
and now once we've got all these region proposals then because we're processing them all sort of in a shared way by sharing these expensive convolutions across the entire image that we can process all of these region proposals in less than a second altogether.
So fast R-CNN ends up being bottlenecked by just the computing of these region proposals.
Thankfully we've solved this problem with faster R-CNN.
So the idea in faster R-CNN is to just make, so the problem was the computing the region proposals using this fixed function was a bottleneck.
So instead we'll just make the network itself predict its own region proposals.
And so the way that this sort of works is that again, we take our input image, run the entire input image altogether through some convolutional layers to get some convolutional feature map representing the entire high resolution image
and now there's a separate region proposal network
which works on top of those convolutional features and predicts its own region proposals inside the network.
Now once we have those predicted region proposals then it looks just like fast R-CNN
where now we take crops from those region proposals from the convolutional features, pass them up to the rest of the network.
And now we talked about multi-task losses and multi-task training networks to do multiple things at once.
Well now we're telling the network to do four things all at once so balancing out this four-way multi-task loss is kind of tricky.
But because the region proposal network needs to do two things: it needs to say for each potential proposal is it an object or not an object, it needs to actually regress
the bounding box coordinates for each of those proposals,
and now the final network at the end needs to do these two things again.
Make final classification decisions for what are the class scores for each of these proposals,
and also have a second round of bounding box regression to again correct any errors that may have come from the region proposal stage.
Question?
So the question is that sometimes multi-task learning might be seen as regularization and are we getting that affect here?
I'm not sure if there's been super controlled studies
on that but actually in the original version of the faster R-CNN paper they did a little bit of experimentation like what if we share
the region proposal network, what if we don't share?
What if we learn separate convolutional networks for the region proposal network versus the classification network?
And I think there were minor differences but it wasn't a dramatic difference either way.
So in practice it's kind of nicer to only learn one because it's computationally cheaper.
Sorry, question?
Yeah the question is how do you train this region proposal network because you don't know, you don't have ground truth region proposals for the region proposal network.
So that's a little bit hairy. I don't want to get too much into those details
but the idea is that at any time you have a region proposal which has more than some threshold of overlap with any of the ground truth objects
then you say that that is the positive region proposal and you should predict that as the region proposal
and any potential proposal which has very low overlap with any ground truth objects should be predicted as a negative.
But there's a lot of dark magic hyperparameters in that process and that's a little bit hairy.
Question?
Yeah, so the question is what is the classification loss on the region proposal network and the answer is
that it's making a binary, so I didn't want to get into too much of the details of that architecture 'cause it's a little bit hairy but it's making binary decisions.
So it has some set of potential regions that it's considering and it's making a binary decision for each one.
Is this an object or not an object?
So it's like a binary classification loss.
So once you train this thing then faster R-CNN ends up being pretty darn fast.
So now because we've eliminated this overhead from computing region proposals outside the network,
now faster R-CNN ends up being very very fast compared to these other alternatives.
Also, one interesting thing is that because we're learning the region proposals here you might imagine
maybe what if there was some mismatch between this fixed region proposal algorithm and my data?
So in this case once you're learning your own region proposals then you can overcome that mismatch if your region proposals are somewhat weird or different than other data sets.
So this whole family of R-CNN methods, R stands for region, so these are all region-based methods
because there's some kind of region proposal and then we're doing some processing, some independent processing for each of those potential regions.
So this whole family of methods are called these region-based methods for object detection.
But there's another family of methods that you sometimes see for object detection
which is sort of all feed forward in a single pass.
So one of these is YOLO for You Only Look Once.
And another is SSD for Single Shot Detection
and these two came out somewhat around the same time.
But the idea is that rather than doing independent processing for each of these potential regions instead we want to try to treat this like a regression problem and just make
all these predictions all at once with some big convolutional network.
So now given our input image you imagine dividing that input image into some coarse grid, in this case it's a seven by seven grid
and now within each of those grid cells you imagine some set of base bounding boxes.
Here I've drawn three base bounding boxes like a tall one, a wide one, and a square one but in practice you would use more than three.
So now for each of these grid cells and for each of these base bounding boxes you want to predict several things.
One, you want to predict an offset off the base bounding box to predict what is the true location of the object off this base bounding box.
And you also want to predict classification scores so maybe a classification score for each of these base bounding boxes.
How likely is it that an object of this category appears in this bounding box.
So then at the end we end up predicting from our input image, we end up predicting this giant tensor of seven by seven grid by 5B + C.
So that's just where we have B base bounding boxes, we have five numbers for each giving our offset and our confidence for that base bounding box
and C classification scores for our C categories.
So then we kind of see object detection as this input of an image, output of this three dimensional tensor
and you can imagine just training this whole thing with a giant convolutional network.
And that's kind of what these single shot methods do
where they just, and again matching the ground truth objects into these potential base boxes becomes a little bit hairy but that's what these methods do.
And by the way, the region proposal network that gets used in faster R-CNN ends up looking quite similar to these where they have some set
of base bounding boxes over some gridded image, another region proposal network does some regression plus some classification.
So there's kind of some overlapping ideas here.
So in faster R-CNN we're kind of treating the object, the region proposal step as kind of this fixed end-to-end regression problem and then we do the separate per region processing but now with these single shot methods
we only do that first step and just do all of our object detection with a single forward pass.
So object detection has a ton of different variables.
There could be different base networks like VGG,
ResNet, we've seen different metastrategies for object detection including this faster R-CNN
type region based family of methods,
this single shot detection family of methods.
There's kind of a hybrid that I didn't talk about called R-FCN which is somewhat in between.
There's a lot of different hyperparameters
like what is the image size, how many region proposals do you use.
And there's actually this really cool paper that will appear at CVPR this summer that does a really
controlled experimentation around a lot of these different variables and tries to tell you how do these methods all perform under these different variables.
So if you're interested I'd encourage you to check it out
but kind of one of the key takeaways is that the faster R-CNN style of region based methods tends to give higher accuracies but ends up being
much slower than the single shot methods
because the single shot methods don't require this per region processing.
But I encourage you to check out this paper if you want more details.
Also as a bit of aside, I had this fun paper with Andre a couple years ago that kind of combined object detection with image captioning
and did this problem called dense captioning
so now the idea is that rather than predicting a fixed category label for each region,
instead we want to write a caption for each region.
And again, we had some data set that had this sort of data where we had a data set of regions together with captions
and then we sort of trained this giant end-to-end model that just predicted these captions all jointly.
And this ends up looking somewhat like faster R-CNN where you have some region proposal stage
then a bounding box, then some per region processing.
But rather than a SVM or a softmax loss instead those per region processing has a whole RNN language model that predicts a caption for each region.
So that ends up looking quite a bit like faster R-CNN.
There's a video here but I think we're running out of time so I'll skip it.
But the idea here is that once you have this, you can kind of tie together a lot of these ideas
and if you have some new problem that you're interested in tackling like dense captioning,
you can recycle a lot of the components that you've learned from other problems like object detection and image captioning
and kind of stitch together one end-to-end network that produces the outputs that you care about for your problem.
So the last task that I want to talk about is this idea of instance segmentation.
So here instance segmentation is in some ways like the full problem
We're given an input image and we want to predict one, the locations and identities of objects in that image similar to object detection, but rather than just
predicting a bounding box for each of those objects, instead we want to predict a whole segmentation mask
for each of those objects and predict which pixels in the input image corresponds to each object instance.
So this is kind of like a hybrid between semantic segmentation and object detection
because like object detection we can handle multiple objects and we differentiate the identities of different instances so in this example
since there are two dogs in the image and instance segmentation method actually distinguishes between the two dog instances
and the output and kind of like semantic segmentation we have this pixel wise accuracy where for each of these objects we want to say which pixels belong to that object.
So there's been a lot of different methods that people have tackled, for instance segmentation as well,
but the current state of the art is this new paper called Mask R-CNN that actually just came out on archive about a month ago so this is not yet published, this is like super fresh stuff.
And this ends up looking a lot like faster R-CNN.
So it has this multi-stage processing approach
where we take our whole input image, that whole input image goes into some convolutional network and some learned region proposal network that's exactly the same as faster R-CNN
and now once we have our learned region proposals then we project those proposals onto our convolutional feature map just like we did in fast and faster R-CNN.
But now rather than just making a classification and a bounding box for regression decision for each of those boxes we in addition
want to predict a segmentation mask for each of those bounding box, for each of those region proposals.
So now it kind of looks like a mini, like a semantic segmentation problem inside each of the region proposals that we're getting from our region proposal network.
So now after we do this ROI aligning to warp our features corresponding to the region of proposal into the right shape, then we have two different branches.
One branch will come up that looks exact, and this first branch at the top looks just like faster R-CNN and it will predict classification scores
telling us what is the category corresponding to that region of proposal or alternatively whether or not it's background.
And we'll also predict some bounding box coordinates that regressed off the region proposal coordinates.
And now in addition we'll have this branch at the bottom which looks basically like a semantic segmentation mini network which will classify for each pixel
in that input region proposal whether or not it's an object
so this mask R-CNN problem, this mask R-CNN architecture just kind of unifies all of these different problems that we've been talking about today into one nice jointly end-to-end trainable model.
And it's really cool and it actually works really really well so when you look at the examples in the paper they're kind of amazing.
They look kind of indistinguishable from ground truth.
So in this example on the left you can see that there are these two people standing in front of motorcycles, it's drawn the boxes around these people, it's also gone in and labeled all the pixels of those people and it's really small
but actually in the background on that image on the left there's also a whole crowd of people standing very small in the background.
It's also drawn boxes around each of those and grabbed the pixels of each of those images.
And you can see that this is just, it ends up working really really well and it's a relatively simple addition on top of the existing faster R-CNN framework.
So I told you that mask R-CNN unifies everything we talked about today and it also does pose estimation by the way.
So we talked about, you can do pose estimation by predicting these joint coordinates for each of the joints of the person
so you can do mask R-CNN to do joint object detection, pose estimation, and instance segmentation.
And the only addition we need to make is that for each of these region proposals we add an additional little branch
that predicts these coordinates of the joints for the instance of the current region proposal.
So now this is just another loss, like another layer that we add, another head coming out of the network and an additional term in our multi-task loss.
But once we add this one little branch then you can do all of these different problems jointly and you get results looking something like this.
Where now this network, like a single feed forward network
is deciding how many people are in the image, detecting where those people are, figuring out the pixels corresponding to each
of those people and also drawing a skeleton estimating the pose of those people and this works really well even in crowded scenes like this classroom where there's a ton of people sitting and they all overlap each other and it just seems to work incredibly well.
And because it's built on the faster R-CNN framework it also runs relatively close to real time
so this is running something like five frames per second on a GPU because this is all sort of done in the single forward pass of the network.
So this is again, a super new paper but I think that this will probably get a lot of attention in the coming months.
So just to recap, we've talked.
Sorry question?
The question is how much training data do you need?
So all of these instant segmentation results were trained on the Microsoft Coco data set
so Microsoft Coco is roughly 200,000 training images. It has 80 categories that it cares about
so in each of those 200,000 training images it has all the instances of those 80 categories labeled.
So there's something like 200,000 images for training and there's something like I think an average of fivee or six instances per image. So it actually is quite a lot of data.
And for Microsoft Coco for all the people in Microsoft Coco they also have all the joints annotated as well so this actually does have quite a lot
of supervision at training time you're right, and actually is trained with quite a lot of data.
So I think one really interesting topic to study moving forward is that we kind of know
that if you have a lot of data to solve some problem, at this point we're relatively confident that you can stitch up some convolutional network that can probably do a reasonable job at that problem
but figuring out ways to get performance like this with less training data is a super interesting and active area of research and I think that's something people will be spending
a lot of their efforts working on in the next few years.
So just to recap, today we had kind of a whirlwind tour of a whole bunch of different computer vision topics
and we saw how a lot of the machinery that we built up from image classification can be applied relatively easily to tackle these different computer vision topics.
And next time we'll talk about, we'll have a really fun lecture on visualizing CNN features.